Minor changes to support benchmarking #138
Conversation
| @@ -179,6 +179,8 @@ def find_text(image_path: str, text: str) -> str: | |||
| 8. DO NOT use try except block to handle the error, let the error be raised if the code is incorrect. | |||
| 9. DO NOT import the testing function as it will available in the testing environment. | |||
| 10. Print the output of the function that is being tested. | |||
| 11. Use the output of the function that is being tested as the return value of the testing function. | |||
| 12. Run the testing function in the end and don't assign a variable to its output. | |||
There was a problem hiding this comment.
Trying to think of ways to keep the list of instructions shorter as I suspect as the list gets longer the ability to follow the directions get's poorer.
Would this also work if we removed instruction 10. (as it seems to accomplish the same thing as 11.) and combine 11. and 12. into "Return the output of the function that is being tested in the test script, do not assign it to another variable" or something shorter?
There was a problem hiding this comment.
I tried a few approaches. This actually is the best prompt so far.
Other approaches significantly increase the chance of the testing function return null.
Here are approaches i have tried:
- Removing 10
- Merge 11 + 12 into one instruction
- Rewrite 11 in a more concise way
- Move 11 + 12 to the 3th and 4th instruction
- Move all instructions up
And some combinations of the above.
From my observation, this is not a problem, but shortening it can actually cause a big problem. i.e. the null values increases from 11% to 80% in my benchmark.
We probably want to rewrite/revisit this entire prompt at some point. It's a bit brittle now.
Minor changes to support benchmarking
get_main_result()toExecutionclass